Leveraging the Short-Term Memory of Hardware to Diagnose Production-Run Software Failures

Failures caused by software bugs are widespread in production runs, causing severe losses for end users. Unfortunately, diagnosing production-run failures is challenging. Existing work cannot satisfy privacy, run-time overhead, diagnosis capability, and diagnosis latency requirements all at once. This paper designs a low overhead, low latency, privacy preserving production-run failure diagnosis system based on two observations. First, short-term memory of program execution is often sufficient for failure diagnosis, as many bugs have short propagation distances. Second, maintaining a shortterm memory of execution is much cheaper than maintaining a record of the whole execution. Following these observations, we first identify an existing hardware performance-monitoring unit, Last Branch Record (LBR), that records the last few taken branches to help diagnose sequential-bug failures. We then propose a simple hardware extension, Last Cache-Coherence Record (LCR), to record the last few cache accesses with specified coherence states and hence help diagnose concurrencybug failures. Finally, we design LBRA and LCRA to automatically locate failure root causes using LBR and LCR. Our evaluation uses 31 real-world sequential and concurrency bug failures from 18 representative open-source software. The results show that with just 16 record entries, LBR and LCR enable our system to automatically locate the root causes for 27 out of 31 failures, with less than 3% run-time overhead. Comparing with existing production-run failure diagnosis systems that rely on sampling, our system does not require a failure to occur many times and significantly shortens the latency of failure diagnosis.